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Surveys allow information to be collected from a sample group and generalized to the 
population at large. Because they are low cost and easy to implement, mail surveys are 
used more frequently for social research than either telephone or face-to-face 
interviews. Those conducting surveys should recognize four potential sources of error - 
sampling error, non-coverage error, non-response error, and measurement error - and 
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take steps to minimize their impact. Any one of these sources of error may make the 
survey results unacceptable (Groves, 1989; Salantand Dillman, 1994; Dillman,1991 , 
1999). This article describes the four types of errors and summarizes the ways they can 
be reduced. 

SAMPLING ERROR 



Mail surveys, like all surveys, collect information only from the people who are included 
in the sample. Because certain members of the population are deliberately excluded 
through selection of the sample, their responses are not obtained. Conclusions about 
the population at large are thus drawn from sample survey results. The heterogeneity of 
the survey measures among members of the population (in other words, the degree to 
which it does not represent the general population) will cause the so-called sampling 
error. 

Sampling error is examined through inferential statistics applied to sample survey 
results. In general, increasing sample size will decrease sampling error when simple 
random sampling is used. For example, when the sample size is increased from 400 
respondents to 1 ,000 respondents for a simple random sample, the sampling error, 
decreases from 5% to 3%. Survey organizations tend to consider this an acceptable 
trade-off between precision of estimation and costs. Most national polls, for example, 
report a 3% margin of error. When simple random sampling is difficult to conduct, other 
methods, such as cluster sampling and stratification sampling, may be used. Calculating 
estimates of the precision of these methods is complex. 

NON-COVERAGE ERROR 



If some members of the population are not covered by the sampling frame, they have 
no chance of being selected into the sample. It is one of the major reasons that mail 
surveys have not been as useful as desired in surveying the general public. If complete, 
up-to-date lists of populations were available, non-coverage error would not exist. 
However, there are no up-to-date lists that provide complete coverage of all the 
households in the United States. Telephone directories are often out-of-date and also 
don't include the small number of households without a phone. Likewise, driver's license 
lists don't cover all of the population. 

NON-RESPONSE ERROR 



No matter how carefully a sample is selected, some members of the sample simply do 
not respond to the survey questions. When those who respond to the mail survey differ 
on the survey measures from those who don't, non-response error will become a 
problem. A low response rate does not necessarily lead to non-response error. 
However, whether differences exist between the responding and non-responding 
segments of the sample is not known when the survey is conducted. Therefore, low 
response has long been considered the major problem of mail surveys, and the vast 
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majority of research on improving mail survey methods has focused on response rates. 
Research studies have successfully identified methods for improving response rates 
and individual factors associated with improved return rates. Heberlein & Baumgartner 
(1978), for example, used the technique of meta-analysis to test the predictability of 71 
characteristics on response rate. They determined that a ten-variable model predicted 
66% of the variation in the final response rate. Seven of the ten variables were found to 
have positive effect on response rate: 



The number of contacts: 

More contacts will increase the response rate. Advance letters, postcards, follow-up that 
include additional copies of questionnaires, and even telephone calls are all examples 
of such contacts. 



Salience of the topic: 

Questionnaires are more likely to be returned if respondents consider them relevant. A 
very common reason given for non-response is that the survey doesn't mean anything 
to the person who received it. 



Government sponsorship: 

Government-sponsored survey research had higher response rates than that from 
private organizations. Employee population: Samples from some special subgroups, 
such as employees from certain occupations, are more likely to return survey research 
than the general population. 



School or army population: 

Students and military personnel are more likely to return questionnaires than the 
general population. 



Special third contact: 

Following up the advance letter and initial follow-up with the use of special mailing 
procedures, such as certified mail or special delivery, or with personal or telephone 
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contact increases the response rate. 



Incentive on the first contact: 

Incentives included with the first mailing will increase response rate. 



Three factors were found to have negative effect on response rate: 

1. Marketing research sponsorship: Marketing research surveys in which the information 
will benefit the firm have lower response rates. 

2. General population: Samples drawn from the general population have lower response 
rates. 

3. Questionnaire length: Questionnaires with more items or more pages have a lower 
return rate. 

Goyder (1982) replicated this study with similar results, except that the negative effect 
of market research sponsorship disappeared. Church (1993), using meta-analysis, 
tested the effects of four types of incentives-monetary (cash and check) and 
non-monetary (entrance to lottery, donation to charity, coffee, books, pens, key rings, tie 
clip, golf balls, stamps etc.) incentives mailed with the survey and monetary and 
non-monetary incentives given upon the return of the questionnaire. His findings 
demonstrated meaningful increases in response rates only for the two initial mailing 
incentive conditions and not for those where the incentives were made contingent on 
return response. Further, no statistically significant difference was found between 
monetary and non-monetary incentives. Eichner & Habermehl (1981), using studies 
from Austria and West Germany, suggested potential cross-cultural differences. In 
contrast to Americans, the European data suggested that government sponsorship has 
negative effect on final response rate, while general population and questionnaire length 
have positive effects. 

Fox, Crask and Kim (1988), using a different meta-analysis method, identified the 
following six methods of improving response rate. There is little or no interaction effect 
among these factors: 

* University sponsorship (vs. business sponsorship) 

* Pre-notification by letter 

* Stamped return postage (vs. business reply) 
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* Postcard follow-up, first-class (vs. second-class and bulk) outgoing postage 

* Green questionnaire (vs. white questionnaire) 

* A small monetary incentive 

Armstrong & Luske's research (1987) also shows a positive effect for applying postage 
to a return letter (vs. including business-reply envelopes). 

MEASUREMENT ERROR 



Unlike sampling error, non-coverage error, and non-response error, which arise from 
non-observations or non-participation, measurement error results from mistakes made 
by respondents. Measurement error results when respondents fill out surveys, but do 
not respond to specific questions, or provide inadequate answers to open-ended 
questions, or fail to follow instructions telling them to skip certain sections depending on 
their answers to previous questions. Measurement errors also arise from lack of control 
of the sequence in which the questions were asked, and various respondents' 
characteristics. These problem areas tend to be balanced by two advantages of mail 
surveys: the absence of an interviewer lessens the likelihood both of respondents' 
feeling driven to provide socially desirable response and of interviewers' accidental or 
purposeful subversion of the purpose of the survey (Dillman,1 978). 

Mixed-mode surveys introduce new considerations related to measurement error. The 
issue becomes not only how accurate the data obtained are, but also whether the 
answers are the same as those obtained for telephone, Internet, and face-to-face 
interview surveys. There is evidence that some differences exist between responses to 
certain questions asked by mail versus by telephone or face-to-face interview surveys: 

* Order effects were less likely to occur in mail surveys than telephone surveys (Bishop 
et al., 1988). In other words, which questions are asked first appears to influence 
respondents more during telephone surveys than mail surveys. 

* Telephone and face-to-face respondents tend to select more extreme answers than 
mail respondents when vaguely quantified scale categories are used. Mail respondents 
ten to distribute themselves across the full scale (Hochstim, 1967; Mangione et al., 

1982; Walker & Restuccia 1984). 

* Mail surveys are more reliable than telephone and face-to-face interview surveys 
(DeLeeuw, 1992). 

Potential explanations for these differences have been suggested, but each one can 
explain only part of the differences across survey methods. Most of the comparative 
studies have been empirically focused and have made only very limited attempts to 
provide theoretical explanations for the differences. More studies are needed to develop 
a theory of response effect. 
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SUMMARY 



During past years, tremendous progress has been made in improving survey response 
rates. Measurement error issues have also been identified. The increasing interest in 
mixed-mode surveys will likely lead to more focused attention on measurement error 
issues. Reducing measurement error will be an important advance for this method of 
social science research. 
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